Migration Utility

We ship a migration utility: a set of Databricks notebooks and Python modules that read your workspace, move your metastore, and prepare your jobs for Yeedu. It's the thing that turns a migration from a quarter of manual archaeology into a report you can act on.

It runs in three phases.

flowchart LR
    A[Phase 1<br/>Hive Metastore to Unity Catalog] --> B[Phase 2<br/>Collect environment inventory]
    B --> C[Phase 3<br/>Migrate jobs to Yeedu]

Phase	Directory	What it does
1	01_hms_migration/	Migrates Hive Metastore tables, views, and permissions into Unity Catalog
2	02_metadata_collection/	Reads system.information_schema and writes a CSV inventory of tables, functions, volumes, and dependency edges
3	03_databricks_to_yeedu/	Downloads job files, rewrites Databricks-specific code, validates access, and generates Airflow DAGs

Phases 1 and 2 are described here. Phase 3 is large enough to have its own page, at Job Migration.

warning

Account Administrator permissions in Databricks are needed to complete the migration. Phase 1 additionally needs Metastore Admin and Workspace Admin, plus CREATE CATALOG and CREATE EXTERNAL LOCATION.

Phase 1: Hive Metastore to Unity Catalog

Plenty of Databricks estates still have a Hive Metastore underneath them, full of managed tables pointing at /mnt/something. Unity Catalog wants real storage URLs. Phase 1 closes that gap.

Three notebooks, run in order. The first discovers your mount points and writes the mapping from /mnt/datalake to abfss://container@storage.dfs.core.windows.net/path. The second scans every Hive Metastore table and lists the storage locations not yet registered as Unity Catalog External Locations. The third does the migration.

A note on compute, because it trips people up: step one has to run on Personal Compute, since Databricks restricts mount-point discovery to it, while steps two and three have to run on Shared Compute, since that's what Hive Metastore RBAC operations require.

The one manual loop

Step two writes a CSV with a FILL_IN_CREDENTIAL_NAME placeholder for every storage location it finds.

adls_path,location_name,credential_name,comment
abfss://container@storage.dfs.core.windows.net/data,data_location,FILL_IN_CREDENTIAL_NAME,

You replace each placeholder with a real Databricks storage credential name, re-run the notebook, and repeat until it reports a credential for every storage account location; credential names live under Settings → Credentials in the Databricks workspace, and each one you name has to already exist there with the permissions its storage location needs, which is precisely why we don't automate this step. Guessing which credential is meant to own which path is a decision about your security posture, not a lookup, and a tool that guessed wrong would hand you a migration that appears to succeed while writing data through the wrong identity.

We ask instead.

What happens to each object type

Object type	Behaviour
Managed tables	Data is copied with CREATE TABLE AS SELECT into the configured ABFSS location, then row counts are compared between source and target
Managed tables, with RECREATE_HMS_MANAGED=true	If row counts match, the original Hive Metastore table is dropped and replaced with an external table pointing at the Unity Catalog storage location
Managed tables, with RECREATE_HMS_MANAGED=false	The original is left alone. Two copies of the data then exist, and you need a synchronisation plan
External tables	Mount paths are replaced with real storage URLs. The Unity Catalog table points at the same location the Hive Metastore table did
Views	A view whose underlying tables haven't been migrated is skipped. Resolve the dependency and re-run

Row counts are compared, always. That's the safety net on the managed-table copy, and it's why RECREATE_HMS_MANAGED=true won't drop anything unless the counts agree.

What Phase 1 produces

File	Contents
uc_migration_object_grants.sql	Grants that reproduce the Hive Metastore access controls in Unity Catalog
migration_log_[timestamp].csv	Per-object migration status: migrated, partial, error, or skipped
summary_report	Migration summary with reasons and next steps

RBAC is configurable. With MOVE_RBAC=true the object-level grants are applied during migration; with false they're written to uc_migration_object_grants.sql for you to review and run yourself. Teams with a change-control process usually want the second.

Phase 2: Collect environment inventory

Phase 2 reads system.information_schema and writes the inventory that Phase 3 depends on for object lookups and dependency resolution. It needs USE CATALOG on the system catalog and nothing else.

File	Contents
tables.csv	All tables, views, and materialized views
functions.csv	All registered functions, including UDFs
volumes.csv	All volumes
view_edges.csv	View dependency relationships
function_edges.csv	SQL function dependency relationships
edges_all.csv	All dependency edges, consolidated
mounts.csv	DBFS mount points, optional

view_edges.csv is the file that earns its keep. Phase 3 walks it so that an access check on a view becomes an access check on the view plus everything underneath it.

ADLS credentials

Phase 3 validates that the target principal can actually reach the ADLS Gen2 storage behind your tables, and it reads service principal credentials from adls_credentials.csv. The utility creates a row for every storage account and container it discovers, with UPDATE_THIS placeholders in the client_id and tenant_id columns.

Rows still holding placeholders are skipped during validation and reported.

The client_secret column takes an XOR-encrypted value, produced by a helper the utility ships at 03_databricks_to_yeedu/bin/encrypt_secret.py. Encrypted values are detected and decrypted automatically during validation, so there's nothing else to configure.

warning

Never commit adls_credentials.csv to a git repository. XOR encryption with a fixed key is obfuscation, not cryptography: it keeps plaintext secrets out of a CSV, and it does not make the file safe to share. Restrict access to the utility's inventory and output directories.

What you need in place

Area	Requirement
Yeedu	Username, REST API URL, and a Personal Access Token
Yeedu	A cluster with the Databricks metastore attached, so it can read Unity Catalog
Airflow	An HTTP connection named yeedu_conn pointing at your Yeedu REST API, and a variable named yeedu_token holding your Yeedu token
Databricks	Workspace URL, a Personal Access Token, and a target catalog such as hms_uc
Storage	An ADLS Gen2 account reachable from both Databricks and Yeedu, registered as a Unity Catalog External Location

Every phase takes a UTILITY_BASE_PATH parameter, which is the directory the phases use to read each other's output.

warning

UTILITY_BASE_PATH must be identical across all three steps of Phase 1 and across Phases 2 and 3. Phases that disagree about it will not find each other's files.

Full setup detail lives in the Yeedu Airflow Operator prerequisites and in the utility's own README.